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Abstract 


This  research  focuses  on  identification  and  description  of  patterns  used  in  Test  and 
Evaluation  (T&E)  operations.  Patterns  are  defined  as  named  and  well-known 
problem/solution  pairs  that  can  be  applied  in  new  contexts,  with  advice  on  how  to  apply  it 
in  a  novel  situations  and  discussion  of  its  trade-offs,  implementations,  and  variations. 
They  have  been  well  established  and  used  in  software  engineering,  but  have  only  recently 
been  considered  for  systems  engineering.  In  the  Department  of  Defense  (DOD),  T&E  is 
required  for  all  acquisitions  prior  to  transitioning  into  operations.  As  identified  and 
described  in  this  thesis,  each  of  the  patterns  fits  in  one  of  the  three  categories:  behavioral, 
creational,  and  functional.  The  research  has  identified  eight  named  patterns  to  include: 
primary  pattern,  objective  pattern,  cost  estimating  pattern,  scheduling  pattern,  risk 
management  pattern,  work  acceptance  process  pattern,  hazard  control  pattern,  and  test 
team  pattern.  The  implementation  of  patterns  by  Department  of  Defense  is  expected  to 
alleviate  some  of  the  common  challenges  encountered  during  T&E,  provide  a  consistent 
higher  quality  and  expectation,  as  well  as  efficiently  use  test  infrastructure  and  personnel 
resources. 
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Identifying  System  Patterns  to  Resolve  Challenges  in  the  Test  and  Evaluation 

Operation 


I.  INTRODUCTION 


General  Issue 

This  research  work  concentrates  on  identifying  and  describing  some  of  the 
patterns  and  challenges  found  in  Test  and  Evaluation  (T&E)  operations.  As  described  in 
(Lannan,  2004),  patterns  are  named  and  well-known  problem/solution  pairs  that  can  be 
applied  in  new  contexts,  with  advice  on  how  to  apply  it  in  a  novel  situations  and 
discussion  of  its  trade-offs,  implementations,  and  variations.  Implementation  of  patterns 
in  T&E  operations  by  Department  of  Defense  T&E  communities  is  expected  to  alleviate 
or  eliminate  some  of  the  common  challenges  encountered  during  the  T&E  operations. 


Research  Objectives/Questions 

Patterns  have  been  in  use  in  software  development  community  for  a  while. 
Contrary,  the  use  of  patterns  in  T&E  processes  has  not  been  common.  The  objective  of 
this  thesis  is  to  identify  and  describe  the  T&E  patterns  that  can  be  used  to  alleviate  or 
eliminate  some  of  the  challenges  encountered  during  T&E  operations.  To  accomplish 
thesis  objective,  the  following  research  questions  in  Table  1  will  be  investigated. 


Table  1.  Research  Objectives  and  Questions 


Objective 

Research  Question 

Identify  challenges  encountered  during 

T&E  Operation 

How  to  identify  some  of  the  challenges 
encountered  by  a  typical  T&E 
organization? 

Detennine  and  discuss  the  patterns  utilized 
in  T&E  Operation 

How  to  detennine  the  some  of  the  T&E 
patterns  that  organizations  can  use  to  plan, 
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execute,  and  report  before,  during,  and 
after  T&E  event? 

Describe  the  use  of  patterns  to  resolve 
challenges  encountered  during  T&E 
Operation 

How  could  the  T&E  communities  use  the 
patterns  to  overcome  challenges 
encountered  during  T&E  operations? 

Methodology 

Through  observation  and  review  of  project  documentation,  patterns  in  T&E 
operations  can  be  identified.  This  thesis  uses  the  observation  and  review  methodology  to 
detennine  the  patterns  that  are  common  in  T&E  operations  and  have  never  been 
identified  and  implemented  as  patterns.  Additionally,  through  observation  and  review, 
some  of  challenges  encountered  during  T&E  operations  are  identified  and  described. 
Finally,  an  association  of  challenges  to  patterns  will  be  conducted  showing  how  the 
patterns  could  help  alleviate  or  eliminate  the  impact  of  the  associated  challenges. 


Assumptions 

The  following  assumptions  are  used  in  this  thesis. 


(1)  T&E  patterns  are  inherently  used  in  T&E  environment,  like  patterns  of 
design  are  used  in  engineering 

(2)  The  T&E  patterns  should  address  problems/challenges  and  be 
applicable  to  all  the  projects  and  systems  under  test 

(3)  There  are  cost,  quality,  and  schedule  benefits  for  using  T&E  patterns 

(4)  Patterns  can  be  identified  and  described 

(5)  Only  patterns  for  DOD  T&E  Operation  are  considered 

(6)  Investigation  of  T&E  patterns  for  systems  and  processes  are  for 
unclassified  information  only 
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Implications 

T&E  patterns  are  applied  across  the  DOD  T&E  organizations  to  avoid  costly 
processes,  reduce  schedule  risks,  and  achieve  better  results.  Since  there  are  numerous 
T&E  projects  in  the  DOD  supporting  different  systems,  T&E  organizations  have  adopted 
patterns  that  suit  their  T&E  requirements. 
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II.  LITERATURE  REVIEW 


Chapter  Overview 

This  chapter  presents  the  literature  review  on  the  previous  work  on  the  same  topic 
or  similar  research  conducted.  As  described  in  (Creswell,  2002),  literature  review 
accomplishes  several  purposes.  In  this  thesis  the  literature  applicable  to  T&E  Operation 
in  the  DOD  and  System  and  software  patterns  will  be  covered. 

Test  and  Evaluation 

T&E  is  the  process  by  which  a  system  or  components  are  compared  against 
requirements  and  specifications  through  testing.  The  results  are  evaluated  to  assess 
progress  of  design,  perfonnance,  and  supportability  (Claxton,  2005).  According  to 
(USD(AT&L),  2007)  the  test  and  evaluation  shall  be  integrated  throughout  the  defense 
acquisition  process.  Structured  test  and  evaluation  provides  essential  information  to  decision¬ 
makers,  assess  attainment  of  technical  performance  parameters,  and  determines  whether 
systems  are  operationally  effective,  suitable,  survivable,  and  safe  for  intended  use.  This 
directive  mandates  the  conduct  of  test  and  evaluation,  integrated  with  modeling  and 
simulation,  to  facilitate  learning,  assess  technology  maturity  and  interoperability,  facilitate 
integration  into  fielded  forces,  and  confirm  performance  against  documented  capability  needs 
and  adversary  capabilities.  As  described  in  (USD(AT&L),  2008),  the  fundamental  purpose  of 
T&E  is  to  provide  knowledge  to  assist  in  managing  the  risks  involved  in  developing, 
producing,  operating,  and  sustaining  systems  and  capabilities.  T&E  measures  progress  in 
both  system  and  capability  development.  T&E  provides  knowledge  of  system  capabilities  and 
limitations  to  the  acquisition  community  for  use  in  improving  the  system  performance,  and 
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the  user  community  for  optimizing  system  use  in  operations.  T&E  expertise  must  be  brought 
to  bear  at  the  beginning  of  the  system  life  cycle  to  provide  earlier  learning  about  the  strengths 
and  weaknesses  of  the  system  under  development.  The  goal  is  early  identification  of 
technical,  operational,  and  system  deficiencies,  so  that  appropriate  and  timely  corrective 
actions  can  be  developed  prior  to  fielding  the  system.  The  DOD  employs  three  formal  types 
of  T&E  in  the  acquisition  of  weapon  systems:  Developmental  Test  and  Evaluation 
(DT&E),  Operational  Test  and  Evaluation,  and  Life-Fire  Test  and  Evaluation  (LFT&E) 
(Defense  Acquisition  Guide,  2012).  The  DT&E  is  the  disciplined  process  of  generating 
experimental  performance  data  from  systems,  subsystems,  components  and  materiel  for 
the  purpose  of  informing  optimum  solutions  and  the  state  of  performance  progress  toward 
design  performance  goals.  The  OT&E  determines  the  operational  effectiveness  and 
operational  suitability  of  a  system  under  realistic  operational  conditions,  including  joint 
combat  operations;  determines  the  satisfaction  of  thresholds  in  the  approved  JCIDS 
documents  and  critical  operational  issues;  assesses  impacts  to  combat  operations;  and 
provides  additional  information  on  the  system’s  operational  capabilities.  The  LFT&E 
objectives  provide  a  timely  assessment  of  the  vulnerability/lethality  of  a  system  as  it 
progresses  through  its  design  and  development,  prior  to  full-rate  production.  The  Test  and 
Evaluation  (T&E)  process  is  an  integral  part  of  the  Systems  Engineering  Process  (SEP), 
which  identifies  levels  of  performance  and  assists  the  developer  in  correcting  deficiencies 
(Defense  Acquisition  University,  2005). 

In  (Fabrycky,  2006)  the  general  T&E  process  as  taught  in  Systems  Engineering 
Schools  such  as  Air  Force  Institute  of  Technology  (AFIT)  is  presented.  Two  steps  were 
discussed  during  the  course.  Firstly,  the  preparation  for  system  T&E  is  conducted.  This 
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step  commences  after  the  initial  planning,  and  prior  to  the  start  of  fonnal  evaluation.  The 
preparation  includes  selection  of  test  item(s),  establishment  of  test  procedures,  selection 
of  test  site,  selection  and  training  of  test  personnel,  preparation  of  test  facilities  and 
resources,  the  acquisition  of  support  equipment,  and  test  supply  support.  Second  step  is 
conducting  system  test,  data  collection,  and  test  reporting.  During  this  step,  the  system  is 
tested  per  test  plan  and  data  collected  and  analyzed  which  leads  to  an  assessment  of 
system  performance  and  effectiveness  characteristics. 

Challenges  Encountered  During  T&E  Operations 

The  following  are  some  of  the  challenges  experienced  by  the  author  and  personnel 
in  T&E  community.  The  identification  of  the  challenges  was  by  review  of  past  T&E 
documentation  and  informal  input  from  personnel  in  the  T&E  community.  These 
challenges,  treated  as  risks,  can  have  serious  impact  on  project  schedule,  cost,  and  system 
quality.  Patterns  are  developed  to  alleviate  the  impact  of  or  eliminate  the  challenges. 

Cost  Overruns 

Cost  overruns  usually  occur  due  to  limited  budget,  scope  creep,  and  poor  project 
management.  Due  to  requirements  to  conduct  T&E  processes  for  all  DOD  systems, 
limited  funds  are  distributed  for  each  item  resulting  in  risk  of  scheduling  to  do  too  much 
with  too  little.  In  these  situations  there  are  always  risks  for  cost  overruns.  Usually  the 
funds  budgeted  for  the  T&E  process  are  exhausted  before  the  project  is  completed.  If  the 
sponsor  cannot  provide  more  funds  to  complete  the  project  then  the  T&E  organization 
either  uses  the  funds  for  operating  costs  to  complete  the  project  or  just  terminates  the 
project.  Tennination  of  a  project  could  result  in  a  bad  reputation  to  the  testing 
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organization  resulting  in  the  loss  of  future  project  opportunities.  Additionally,  using 
funds  budgeted  for  other  projects  or  project  tasks  for  T&E  purposes  can  impact  the 
outcome  of  other  projects  or  the  health  of  organization.  The  cost  challenges  can  be 
alleviated  by  proper  project  management  that  includes  better  budgeting  of  funds  and 
prioritizing  of  events  conducted.  A  Cost  Estimating  process  that  accounts  for  all  project 
costs  is  used  to  reduce  the  risk  of  insufficient  funds  to  a  project.  The  major  cost  elements 
in  the  cost  estimate  are  labor  hours,  travel,  and  equipment/material  procurement  costs. 
Safety  Hazards 

T&E  operation  safety  hazards  pose  the  danger  of  personnel  injury  and/or 
equipment  damage  when  the  mishaps  occur.  Some  of  the  common  safety  hazards  during 
T&E  operation  include  accidental  ammunition  initiation  during  inspection,  laser  injury 
during  laser  operations,  cuts  and  bruises  during  equipment  setup,  and  chemical  bum  or 
poisoning  during  explosive  lab  testing.  Prior  to  commencement  of  a  T&E  operation,  all 
potential  hazards  are  identified  and  mitigation  procedures  implemented  to  reduce  the 
probability  of  the  mishap  happening  and  its  effect  if  it  happens.  Table  2  is  a  sample 
hazard  analysis  table  found  in  (CRANE,  2009)  that  shows  recommended  steps  to  reduce 
the  probability  and  impacts  of  mishaps.  Mishaps  are  likely  to  result  to  death  or  serious 
injury  to  personnel  or  damage  to  test  equipment.  A  serious  mishap  resulting  in  death  or 
serious  injury  to  personnel  could  have  serious  damage  to  an  organization  from  bad 
reputation  to  shutting  down. 


Table  2.  Hazard  Analysis 


Procedure: 


Branch: 


Revision  Number: 
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Process 

Step 

Hazard 

Mishap 

Triggering  Event 

Potential 

Mishap 

Prelim  Risk 
Index 

Hazard  Mitigation 
Requirements 

Final  Risk 
Index 

Strict  Schedule 

Time  constraint  in  the  DOD  and  the  need  to  rapidly  provide  systems  to  the  users 
pose  schedule  challenge  to  the  T&E  community.  Timely  analysis  and  reporting  of 
operational  and  live  fire  test  results  are  critical  to  the  DOT&E  mission,  as  well  as  the 
mission  of  every  test  agency.  It  is  essential  that  our  organizations  analyze  test  data  and 
formally  report  on  test  and  evaluation  as  soon  as  practicable,  regardless  of  whether  a 
formal  report  is  required  by  regulation  or  statute.  Test  results  are  most  effective  when 
they  provide  feedback  as  early  as  possible  within  the  acquisition  process.  Delays  in 
reporting  can  prevent  senior  leaders  from  recognizing  serious  problems  that  may 
unnecessarily  risk  the  lives  of  our  Service  members  (Gilmore,  2010).  Usually  the 
developer  provides  the  expected  Period  of  Performance  (PoP)  in  their  Work  Orders  or 
Statement  of  Work  indicating  when  the  work  agreed  upon  is  expected  to  be  completed. 
Given  the  strict  PoP,  the  T&E  agency  determines  the  schedule  that  will  meet  the  PoP 
requirement.  With  multiple  concurrent  projects  in  an  organization  with  similar  resource 
requirements,  the  challenge  is  to  appropriately  schedule  the  right  personnel,  equipment, 
and  facilities  for  the  project  to  ensure  timely  completion  of  project. 

Unskilled  and  Insufficient  Workforce 

DOD  consists  of  diverse  workforce  possessing  different  skills  sets  and  experience 
levels.  For  instance,  there  are  military  veterans  in  the  T&E  community  with  tremendous 
amount  of  experience  on  military  systems  gained  during  their  time  in  service  and  skilled 


8 


personnel  with  advanced  college  degrees.  The  combination  of  experience  and  skills 
produces  a  strong  workforce  to  support  the  T&E  efforts  in  the  DOD.  However,  a  lot  of 
times  both  virtues,  skill  and  experience,  are  required  to  accomplish  most  T&E  tasks.  Lack 
of  the  personnel  possessing  both  virtues  is  a  challenge  since  it  requires  an  organization  to 
hire  two  people  to  perform  work  that  can  be  completed  by  one  person.  In  other  cases, 
there  might  be  too  much  of  one  virtue  and  too  little  of  the  other.  In  this  situation,  there 
will  not  be  sufficient  personnel  to  support  project  tasks  and  the  organizations  are  at  risk 
of  failing  to  meet  sponsor  schedule  requirements.  The  other  challenges  are  trying  to 
integrate  the  DOD’s  diverse  community  into  a  single  unit  to  achieve  the  T&E  objectives. 
Unavailability  of  Test  Equipment 

Test  equipment  includes  the  equipment  set  up  for  data  collection  purposes  and 
support  equipment.  The  data  collection  equipment  includes  but  is  not  limited  to  radar 
systems,  high  speed  video  equipment,  computers,  and  cameras.  The  support  equipment 
includes  but  is  not  limited  to  vehicles,  weather  stations,  conditioning  chambers,  portable 
toilets,  generators,  and  trailers.  The  challenges  experienced  with  test  equipment  are 
unavailability  of  the  appropriate  equipment  due  to  lack  of  funds  to  buy  the  equipment, 
damage  during  testing,  or  the  equipment  being  used  in  another  operation.  In  the  situation 
where  the  equipment  is  unavailable  to  support  the  test,  there  are  always  risks  for  schedule 
delays,  late  delivery  of  customer  products,  loss  of  personnel  to  other  projects,  and  cost 
overrun  due  to  downtime.  The  equipment  risks  can  be  reduced  by  ensuring  that  the 
equipment  is  available  and  ready  for  test  and  is  being  operated  by  trained  personnel 
during  testing. 
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Strict  DOD  Policies 


The  policies  in  place  ensure  that  the  DOD  T&E  efforts  are  conducted  safely 
without  risks  to  the  environment,  testers,  and  general  population.  The  policies  in 
explosive  testing  environment  require  agencies  to  utilize  designated  test  ranges  and  air 
space,  control  the  explosive  residue,  and  perfonn  thorough  cleanup  and  dispose  of  the 
waste  in  designated  hazardous  waste  disposal  areas.  Failure  to  adhere  to  the  policies 
could  lead  to  the  shutting  down  of  the  organization.  Locally,  the  SOP  for  each  project 
contains  the  summary  of  the  policy  requirements  that  need  to  be  met.  The  SECNAVINST 
5000.2E  navy  instruction  (2011)  provides  guidance  on  policies  for  acquisition  of  all 
Department  of  Navy  systems. 

Inclement  Weather 

DOD  organizations  conduct  T&E  operations  throughout  the  year  under  all 
weather  conditions.  The  T&E  projects  are  scheduled  based  on  the  requirements  of  the 
project  and  availability  of  test  resources  including  personnel,  facilities,  funds,  and  time.  A 
project  with  environmental  condition  testing  requirements  such  as  temperature,  wind, 
dust,  and  humidity  are  scheduled  to  be  tested  at  the  sites  with  the  appropriate  weather 
conditions.  For  instance,  test  requiring  cold  temps  would  likely  be  tested  in  winter  times 
or  in  cold  areas  like  Alaska.  Likewise,  a  project  with  humidity  requirements  would  likely 
be  conducted  in  high  humidity  areas  or  during  wet  seasons.  Weather  challenges  are 
encountered  when  there  are  storms  during  testing,  temperatures  too  high  or  too  cold 
posing  health  hazards  to  the  test  team.  Sometimes  storms  are  accompanied  by  lightning. 
Inclement  weather  usually  results  to  shutting  down  the  operations  risking  schedule 
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delays,  cost  overruns,  and  loss  of  key  personnel  and  facilities/equipment  to  other 
scheduled  tests. 

Difficulty  in  Data  Collection 

T&E  is  the  process  of  exercising  the  system  in  accordance  with  the  test  plan  to 
ensure  that  data  is  collected  to  support  the  evaluation  plan.  Test  execution  includes 
executing  a  pilot  and  record  test,  conducting  data  reduction,  validation,  and  analysis,  and 
conducting  a  data  review  to  ensure  data  is  adequate  to  support  evaluation.  Collection  of 
invalid  data  results  in  incorrect  evaluation  of  the  system.  Difficulty  in  collecting  correct 
data  could  be  as  a  result  of  having  wrong  data  collection  equipment,  unskilled  data 
collectors,  poorly  defined  data  collection  requirements,  and  using  faulty  equipment. 

Lack  of  Collaboration 

There  are  many  stakeholders  involved  in  a  T&E  project.  The  major  categories  of 
the  stakeholders  are  the  testing  organization,  system  developing  organization,  and 
program  office.  The  testing  organization  consists  of  the  project  engineers,  data  collectors, 
data  analysts,  equipment  operators,  range  support  personnel,  safety  personnel,  and  subject 
matter  experts.  The  system  developing  organization  may  include  the  contractor  designing 
and  manufacturing  the  system.  The  program  office  consists  of  the  program  manager 
leading  the  effort  of  acquiring  the  system  and  ensuring  successful  transition  to  the 
warfighter.  The  T&E  organization  works  with  all  the  other  stakeholders  during  the 
development  of  the  system  to  transition  to  the  user.  Effective  collaboration  with  all  the 
stakeholders  is  of  paramount  importance  to  ensure  successful  completion  of  the  T&E 
process.  Challenges  occur  when  the  stakeholders  are  not  willing  or  able  to  work  together. 
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It  is  stated  in  (Gilmore,  FY  2012  Annual  Report,  2012)  that  the  lack  of  critically  needed 
collaboration  among  the  technical,  test,  and  requirements  communities  is  not  new.  The 
1986  Packard  Commission  found  that  success  in  new  programs  depends  on  “an  infonned 
trade-off  between  user  requirements,  on  one  hand,  and  schedule  and  cost,  on  the  other”. 
The  participants  could  be  remotely  located  from  each  other  and  reluctant  to  travel  or 
participate  in  teleconferences.  Some  stakeholders  may  not  be  willing  to  accept  necessary 
changes  to  ensure  execution  of  the  T&E  process  resulting  in  delayed  or  cancelled 
projects. 

Unavailability  of  Test  Facilities 

Test  facilities  include  the  test  ranges,  bases,  and  laboratories  used  for  various 
T&E  Operations.  In  February  2012, 1  identified  shortfalls  in  electronic  warfare  test 
resources  that  prevent  adequate  developmental  and  operational  testing  of  many  systems, 
including,  but  not  limited  to,  the  Joint  Strike  Fighter  (Gilmore,  FY  2012  Annual  Report, 
2012). 

Due  to  the  hazardous  nature  of  most  DOD  test  operations,  only  approved  sites  by 
the  government  are  designated  for  testing  conducting  DOD  operations  for  T&E  efforts. 
These  sites  are  remotely  located  inside  the  Continental  United  States  (CONUS)  and 
outside  Continental  United  States  (OCONUS).  Many  of  the  test  facilities  are  also  used  by 
active  military  for  operations  and  exercise.  T&E  organizations  usually  have  access  to 
these  sites  but  only  when  they  are  not  required  for  military  training  exercises.  Any 
schedule  conflict  with  military  exercise  and  T&E  operations  results  in  cancellation  of  the 
T&E  operations  or  postponement  leading  to  delayed  evaluation  of  the  system. 
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Additionally,  too  many  T&E  events  happening  at  the  same  time  results  in  schedule  delay 
or  requirement  to  travel  to  sites  located  far  away  from  the  personnel  supporting  the  test. 
Usually  these  challenges  results  in  increased  costs  and  schedule  delays.  Figure  1  in  shows 
the  ranges  designated  for  DOT  T&E  operations. 


|Major  Range  and  Test  Facility  Base  (MRTFB): 
The  “Critical  Core” 


23  Sites:  Army-8;  Navy-6;  Air  Force-7;  Defense  Agency-2 


Figure  1.  Test  Ranges 


Poor  Project  Management 

Project  management  is  the  application  of  knowledge,  skills,  tools,  and  techniques 
to  project  activities  to  meet  the  project  requirements  (Project  Management  Institute, 
2008).  Usually  the  personnel  leading  T&E  projects  are  usually  engineers  possessing 
limited  or  no  project  management  skills.  Without  necessary  tools  of  project  management, 
it  becomes  quite  challenging  to  successfully  complete  major  T&E  projects.  Some  of  the 
issues  associated  with  poor  management  include  poor  documentation,  poor  scheduling  of 
events,  poor  distribution  of  roles  to  personnel,  and  poor  management  of  funds.  The 
patterns  developed  in  this  research  work  would  help  alleviate  project  challenges  faced  by 
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T&E  communities.  Figure  1  in  (Defense  Acquisition  University,  2005)  shows  test-related 
documentation  that  the  program  manager  is  responsible  for. 


Figure  2.  Test  Related  Documentation 

T&E  is  one  of  the  major  requirements  in  Defense  Acquisition  Management 


System.  Successful  T&E  provides  data  to  the  decision  makers  for  analysis  to  determine  if 


the  DOD  system  is  ready  to  transition  to  the  warfighters.  Failure  to  manage  the  projects 


correctly  could  lead  to  invalid  evaluation  leading  to  delays  or  cancellation  of  programs. 


The  size  and  complexity  of  the  DOD  and  pressure  to  rapidly  transition  systems  to  the 


warfighters  contribute  to  challenges  facing  the  T&E  organizations  during  T&E  execution. 
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The  consequences  of  poor  program  management  include  cost  overruns,  schedule  delays, 
and  delivery  of  poor  quality  products. 

System  and  Software  Design  Patterns 

In  software  engineering,  a  design  pattern  is  a  general  reusable  solution  to  a 
commonly  occurring  problem  within  a  given  context  in  software  design.  It  is  a 
description  or  template  for  how  to  solve  a  problem  that  can  be  used  in  many  different 
situations  (Wikipedia,  2012).  In  (Cloutier  R.  ,  2009)  a  pattern  is  described  as  a  model  or 
facsimile  of  an  actual  thing  or  action,  which  provides  some  degree  of  representation  (an 
abstraction)  to  enable  the  recreation  of  that  entity  over  and  over  again.  He  describes 
systems  engineering/system  architecture  pattern  as  models  which  capture  a  high-level 
structure,  appropriate  to  the  design  of  the  major  components  of  a  system.  Furthermore  in 
(Erl,  2009)  states  that  the  simplest  way  to  describe  a  pattern  is  that  it  provides  a  proven 
solution  to  a  common  problem  individually  documented  in  a  consistent  fonnat  and 
usually  as  part  of  a  larger  collection.  The  notion  of  patterns  originated  with  the  (building) 
architectural  patterns  of  Christopher  Alexander  (Larman,  2004).  Further,  in  the  same 
reference,  it  is  stated  that  patterns  for  software  originated  in  the  1980s  with  Kent  Bent, 
who  became  aware  of  Alexander’s  pattern  work  in  architecture,  and  then  were  developed 
with  Ward  Cunnigham  at  Tektronix. 

In  Center  for  Patterns  and  Systems  Engineering  (Cloutier  R.  ,  2009),  music 
demonstrates  repeating  patterns  to  make  it  easier  to  learn  the  tune.  Three  childhood  songs 
have  the  same  tune  -  Twinkle,  Twinkle  Little  Star;  Baa,  Baa  Black  Sheep  and  The 
Alphabet  Song.  All  are  from  the  same  Mozart  tune.  Mathematics  is  full  of  patterns  - 
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methods  to  solve  similar  problems:  sum  of  squares,  quadratic  equations,  and  standard 
integration  of  common  functions.  The  use  of  cul-de-sacs  by  civil  engineers/architects  in  a 
housing  development  is  another  example  of  a  commonly  found  pattern.  Other  examples 
of  patterns  are  geometric  forms  -  a  large  circle  and  a  small  circle  differ  only  by  the 
radius.  Patterns  have  been  around  in  the  engineering  community  for  quite  some  time. 

Most  notably,  parts  of  the  software  community  have  been  using  them  for  years.  The 
existence  of  patterns  is  almost  universal.  One  can  find  patterns  everywhere  and  the 
human  mind  seems  to  perceive  patterns  without  conscious  thought.  Cloutier  further 
explains  that  patterns  express  relation  between  the  context,  a  problem,  and  a  solution,  and 
document  the  system  attributes  and  usage  guidance.  They  are  time-proven  in  solving 
problems  similar  in  nature  to  the  problem  under  consideration. 

The  notion  of  a  pattern  as  described  in  (Erl,  2009)  is  already  a  fundamental  part  of 
everyday  life.  Thomas  Erl  states  that  without  acknowledging  it  each  time,  we  naturally 
use  proven  solutions  to  solve  common  problems  each  day.  Patterns  in  the  IT  world  that 
revolve  around  the  design  of  automated  systems  are  referred  to  as  design  pattern. 

As  stated  in  (Dong,  Zhao,  &  Sun,  2009)  ,  design  patterns  have  emerged  to  become 
an  important  design  guidance  that  provides  good  generic  solutions  to  recurring  problems. 
Each  design  pattern  documents  a  guideline  in  the  software  development  process  and 
leaves  room  for  application  variations. 

In  (Ambler,  1998),  Scott  described  process  pattern  as  a  collection  of  general 
techniques,  actions,  and/or  tasks  (activities)  for  developing  object-oriented  software. 

Scott  describes  three  types  of  process  patterns  conducted  in  organizations:  Task  process 
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patterns,  Stage  process  patterns,  and  phase  process  patterns.  He  described  task  process 
pattern  as  pattern  that  depicts  the  detailed  steps  to  perform  a  specific  task,  such  as  the 
Technical  Review  and  Reuse  First  process  patterns.  Furthermore,  he  describes  stage 
process  as  the  pattern  that  depicts  the  steps,  which  are  often  performed  iteratively,  of  a 
single  project  stage.  A  project  stage  is  a  higher-level  fonn  of  process  pattern,  one  that  is 
often  composed  of  several  task  process  patterns.  He  describes  phase  process  patterns  as 
pattern  that  depicts  the  interactions  between  the  stage  process  patterns  for  a  single  project 
phase,  such  as  the  Initiate  and  Delivery  phases.  In  Figure  3,  the  program  stage  of  process 
pattern  is  depicted.  Scott  shows  that  there  is  more  to  this  stage  than  simply  writing  source 
code. 


Figure  3.  Process  Pattern 

Design  patterns  are  recurring  solutions  to  software  design  problems  you  find 
again  and  again  in  real-world  application  development.  Patterns  are  about  design  and 
interaction  of  objects,  as  well  as  providing  a  communication  platfonn  concerning  elegant, 
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reusable  solutions  to  commonly  encountered  programming  challenges  (Data  &  Object 
Factory,  LLC,  2001-2013).  The  three  categories  of  the  Gang  of  Four  patterns  generally 
considered  the  foundation  for  all  other  patterns  are  presented  here  as  described  in 
(Gamma,  Helm,  Johnson,  &  Vlissides,  1994).  They  are  categorized  in  three  groups: 
Creational,  Structural,  and  Behavioral. 

Creational  Patterns 

The  creational  patterns  are  ones  that  create  objects  for  you,  rather  than  having  you 
instantiate  objects  directly.  This  gives  your  program  more  flexibility  in  deciding  which 
objects  need  to  be  created  for  a  given  case.  These  include  abstract  factory,  builder, 
factory  method,  prototype,  and  singleton.  Figure  4  below  shows  the  structure  of  the 
builder  pattern.  The  Build  pattern  separates  the  construction  of  a  complex  object  from  its 
representation  so  that  the  same  construction  process  can  create  different  representations. 


Figure  4.  Build  Pattern 

Structural  Patterns 

The  structural  patterns  deal  with  the  composition  of  classes  or  Objects.  The 
structural  patterns  included  Adapter,  Bridge,  Composite,  Decorator,  Facade,  Flyweight, 
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and  Proxy.  As  an  example,  Figure  5  is  presented  here  for  structural  adapter  pattern.  The 
adapter  pattern  converts  the  interface  of  a  class  into  another  interface  clients  expect. 
Adapter  lets  classes  work  together  that  couldn't  otherwise  because  of  incompatible 
interfaces. 


Figure  5.  Adapter  Pattern 


Behavioral  Patterns 

Behavioral  patterns  characterize  the  ways  in  which  classes  or  objects  interact  and 
distribute  responsibility.  Behavioral  patterns  as  identified  and  described  by  GoF  include 
Interpreter,  Template  Method,  Chain  of  Responsibility,  Command,  Iterator,  Mediator, 
Memento,  Observer,  State,  Strategy,  and  Visitor.  The  structural  presentation  of  the 
interpreter  pattern  is  depicted  in  Figure  6.  The  interpreter  pattern  is  described  as  the 
grammar  and  interpretation  of  a  language. 
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Figure  6.  Interpreter  Pattern 


Patterns  in  Enterprise  Software 

In  recent  years  there  has  been  a  small  but  useful  growth  in  describing  patterns  for 
the  development  of  enterprise  systems  (Fowler,  2005).  Two  kinds  of  enterprise  software 
patterns  will  be  discussed:  Patterns  of  Enterprise  Application  Architecture  and  Enterprise 
Integration  Patterns. 

Patterns  of  Enterprise  Application  Architecture 

Martin  Fowler  in  his  Patterns  of  Enterprise  Application  Architecture  provided  a 
list  of  the  patterns  used  in  enterprise  application  architecture.  The  pattern  categories 
discussed  include  Domain  Logic  Patterns,  Data  Source  Architecture  Patterns,  Object- 
Relational  Behavioral  Patterns,  Object-Structural  Patterns,  Web  Presentation  Patterns, 
Distribution  Patterns,  Offline  Concurrency  Patterns,  Session  State  Patterns,  and  Base 
Patterns.  Figure  7,  is  an  example  of  enterprise  application  architecture  pattern  from 
Domain  Logic  Pattern.  It  shows  a  domain  model  pattern  that  creates  a  web  of 
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interconnected  objects,  where  each  object  represents  some  meaningful  individual, 
whether  as  large  as  a  corporation  or  as  small  as  a  single  line  on  an  order  form. 


Figure  7.  Domain  Model  Pattern 


Martin  Fowler  work  concentrated  on  Enterprise  Application  Architecture  in  the 
context  of  a  layered  architecture. 

Enterprise  Integration  Patterns 

Integration  of  messaging  system  is  discussed  in  (Wolfe  &  Hohpe,  2004).  In  this 
book,  integration  means  connecting  computer  systems,  companies,  or  people.  Based  on 
this  meaning,  there  are  six  types  of  integration:  Information  portals,  data  replication, 
shared  business  functions,  service-oriented  architectures,  and  distributed  business 
processes.  Examples  of  patterns  used  in  messaging  systems  are  provided  here.  Messaging 
system  manages  messaging  the  way  a  database  system  manages  data  persistence. 
Messaging  system  is  needed  to  move  messages  from  one  computer  to  another  because 
computers  and  the  networks  that  connect  them  are  inherently  unreliable.  Some  of  the 
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patterns  included  in  messaging  system  include  Request-Reply  pattern  (Figure  8), 
Multiple  Consumers,  Return  Address,  Multiple  Service  Providers  (Figure  9),  Correlation 
Identifier,  and  Pipes- And-Filters. 


Consumer 


r 


Request  Channel 


Request 


Provider 


r 


Reply  Channel 


3 

Reply 


Figure  8.  Request-Reply  pattern 


Figure  9.  Multiple  Service  Providers 


There  are  drawbacks  and  benefits  for  using  patterns.  The  following  are  some  of 
the  benefits  and  drawbacks  identified  in  (Dong,  Zhao,  &  Sun,  2009)  ,  (Cloutier  R.  , 
2009),  and  (Erl,  2009). 

Benefits  for  Using  Design  Patterns 

•  Design  Reuse:  Design  patterns  are  suitable  for  reusing  design  experience  at  a 
high  abstraction  level.  Patterns  allow  the  designers  to  tackle  the  same  problem  by 
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reusing  expert  solutions.  Patterns  enable  reuse  of  good  concepts  and 
implementations,  and  to  preserve  them  for  future  projects. 


•  Adaptability  and  Extensibility:  Software  developers  adopt  patterns  in  their 
design  to  improve  adaptability  and  extensibility. 

•  Documentation  of  Design  Solutions  and  Tradeoffs:  Design  patterns  generally 
document  the  design  decisions  and  tradeoffs  as  well  as  possible  ways  for  future 
evolutions. 


•  Design  Consistency:  Can  be  used  to  ensure  consistency  in  how  systems  are 
designed  and  built.  Consistent  application  tends  to  naturally  improve  the  quality 
of  system  designs. 

•  Design  Standard:  Can  become  the  basis  for  design  standards.  Architectural 
patterns  may  help  control  the  complexity  of  architecture  by  standardizing  it  on  a 
well-known  and  practiced  pattern. 

•  Educational  Aids:  Can  be  used  as  educational  aids  by  documenting  specific 
aspects  of  system  design  (regardless  of  whether  they  are  applied). 

•  Common  Understanding:  Describing  parts  of  the  architecture  in  the  context  of 
known  and  understood  architecture  patterns  results  in  a  common  understanding  of 
the  architecture. 


•  Mitigate  Risks:  Using  and  applying  known  architecture  patterns  in  architecture 
design  introduce  less  risk  than  a  new  architecture  design. 

Drawbacks  for  Using  Design  Patterns 

•  It  is  hard  to  understand  the  systems  since  the  original  design  decisions  and 
tradeoffs  embedded  in  the  design  patterns  are  not  available. 


•  Due  to  lack  of  documentation,  high-level  architecture  and  design  information  is 
often  missing  in  system  implementations,  particularly  for  legacy  systems. 
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•  Over  time,  similar  designs  are  arrived  at  independently  by  different  designers  on 
various  projects. 

•  The  design  reuse  prevents  engineers  from  having  to  reinvent,  or  rediscover  the 
same  concept  and  provides  a  common  vocabulary  for  the  design  concepts  that  a 
project  can  share. 

Summary 

As  opposed  to  other  areas,  patterns  have  been  developed  and  effectively 
implemented  in  software  engineering  field.  The  patterns  studied  were  presented  in  terms 
of  their,  structure,  context,  problem  part,  and  solution  part.  As  detennined  during 
literature  review,  work  has  already  begun  of  identifying  and  applying  patterns  in  other 
areas  including  systems  engineering.  The  literature  review  covered  development  and  use 
of  patterns  mainly  in  software  field.  It  was  observed  that  similar  methods  and  approach  in 
developing  patterns  for  software  applications  can  also  be  used  in  developing  patterns  for 
other  areas  including  T&E  process,  systems  engineering,  and  hardware  design.  The 
benefits  and  drawbacks  of  using  patterns  were  also  identified.  The  literature  review  also 
included  the  identification  of  challenges  commonly  faced  by  T&E  communities.  These 
challenges  were  detennined  through  infonnal  peer  survey  and  review  of  prior  project 
work  at  a  T&E  organization.  The  development  and  appropriate  application  of  identified 
T&E  patterns  leads  to  resolving  the  identified  challenges. 
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III.  METHODOLOGY 


Chapter  Overview 

The  methodology  used  to  develop  patterns  for  T&E  operations  is  described  in  this 
chapter.  Mainly  the  methodology  is  observational  consisting  of  review  of  current  and  past 
T&E  projects  to  identify  and  describe  patterns.  The  patterns  identified  using  this 
methodology  are  presented  in  the  format  shown  in  Table  3.  This  is  the  format  used  in  the 
Design  Patterns  book  (Dougherty,  Sayre,  Seacord,  Svoboda,  &  Togashi,  2009)  to 
describe  the  23  pattern  types  grouped  in  three  pattern  categories:  creational ,  behavioral, 
and  structural. 


Table  3.  Pattern  Elements 


Pattern  Element 

Element  Description 

Pattern  Name  and 
Classification 

A  descriptive  and  unique  name  that  helps  in  identifying  and  referring  to  the 
pattern 

Intent 

Goal  behind  the  pattern  and  the  reason  for  using  it 

Also  Known  As 

Other  names  for  the  pattern 

Motivation  (Forces) 

A  scenario  consisting  of  a  problem  and  a  context  in  which  this  pattern  can  be 
used 

Applicability 

Situations  in  which  this  pattern  is  usable;  the  context  for  the  pattern 

Structure 

A  graphical  representation  of  the  pattern 

Participants 

A  listing  of  the  objects  used  in  the  pattern  and  their  roles  in  the  design. 

Collaboration 

How  objects  used  in  the  pattern  interact  with  each  other 

Consequences 

The  results,  side  effects,  and  tradeoffs  caused  by  using  the  pattern 

Implementation 

Implementation  of  the  pattern;  the  solution  part  of  the  pattern 

Known  Uses 

Examples  of  real  usages  of  the  pattern 

Related  Patterns 

Other  patterns  that  have  some  relationship  with  the  pattern;  discussion  of  the 
differences  between  the  pattern  and  similar  patterns 

(Dougherty,  Sayre,  Seacord,  Svoboda,  &  Togashi,  2009) 


Identification 

Patterns  are  identified  based  on  the  familiarity  of  T&E  projects  to  the  person 
developing  them.  The  experience  of  the  developer  with  T&E  projects  helps  in 
discovering  patterns  in  T&E.  Due  to  repeatability  of  the  processes  conducted  to  achieve 
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T&E  objectives,  over  time  patterns  are  observed  as  the  similar  projects  with  related 
requirements  and  objectives  are  completed.  The  patterns  identified  in  this  thesis  were 
repeatedly  observed  to  occur  to  all  the  projects  conducted  in  the  organization. 
Unfortunately,  the  organization  executing  the  processes  may  not  recognize  them  as 
patterns  resulting  to  wasted  resources  every  time  a  project  is  conducted.  Another  way  to 
the  patterns  are  identified  is  the  review  of  completed  test  plans  and  project  reports  and 
study  of  their  requirements.  The  reports  would  include  consisted  fonnats  indicating  some 
kind  of  pattern  used  in  writing  them.  Additionally,  some  of  the  contents  of  each  test  plan 
or  report  would  be  consistently  similar  to  test  plans  and  reports  of  different  projects 
indicating  their  similarity  to  some  extent  hence  use  of  patterns.  Reviewing  the  test  plans 
and  reports  and  understanding  the  requirements  for  each  project,  patterns  can  be  seen  in 
the  approach  of  accomplishing  the  test  objectives.  For  instance,  test  plans  would  resource 
requirements.  Planning,  execution,  and  reporting  processes  are  the  main  phases  in  T&E 
process.  They  form  the  backbone  of  every  pattern  developed  for  each  T&E  project. 

The  following  main  T&E  processes  consist  of  specific  tasks  and  documentation 
requirements  that  are  included  in  each  of  the  identified  T&E  patterns. 

T&E  Planning 

Operational  Test  (OT)  planning  is  one  of  the  most  important  parts  of  the  OT&E 
process.  Proper  planning  facilitates  the  acquisition  of  data  to  support  the  detennination  of 
the  weapon  system’s  operational  effectiveness  and  suitability.  Planning  must  be  pursued 
in  a  deliberate,  comprehensive,  and  structured  manner.  Careful  and  complete  planning 
may  not  guarantee  a  successful  test  program;  but  inadequate  planning  can  result  in 
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significant  test  problems,  poor  system  perfonnance,  and  cost  overruns  (Claxton,  2005). 
During  T&E  planning,  key  processes  and  documentation  are  completed.  These  processes 
and  documentation  are  completed  either  consecutively  or  concurrently  to  meet 
requirements.  The  documentation  requirements  include  statement  of  work  (SOW),  rough 
order  of  magnitude  (ROM),  test  plan  (TP),  work  unit  assignment  (WUA),  standard 
operation  procedure  (SOP),  safety  releases  (SR),  operational  risk  management  (ORM), 
cost  estimate  (CE)  and  data  collection  Plans  (DCP).  The  processes  include  conducting 
work  acceptance  review  (WAR),  developing  timeline,  developing  operation  plan 
(OpPlan),  coordination  with  all  the  stakeholders,  participating  in  test  readiness  review 
(TRR)  and  content  review  boards  (CRB),  and  identifying  and  reserving  resources 
required  for  the  test. 

T&E  Event  Execution 

An  OT  plan  is  only  as  good  as  the  execution  of  that  plan.  The  execution  is  the 
essential  bridge  between  test  planning  and  test  reporting  (Claxton,  2005).  Prior  to 
commencement  of  event  execution,  the  necessary  test  resources  are  identified  and 
reserved.  These  include  personnel  requirements,  equipment  requirements,  logistics 
requirements,  facilities  requirements,  and  funding  requirements.  In  order  to  be  successful 
in  the  execution  and  meet  sponsor’s  requirements,  the  coordination  of  these  requirements 
requires  a  systematic  approach.  This  approach  is  repeated  many  times  for  each  project  to 
the  point  where  the  personnel  perfonning  it  start  seeing  patterns.  When  a  project  calls  for 
certain  skills,  the  manager  or  project  lead  approaches  the  resource  manager  to  determine 
who  is  well  suited  to  support  the  task.  Likewise,  when  a  project  requires  certain 
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equipment,  the  resource  manager  looks  at  the  equipment  inventory  to  determine  if  the 
equipment  is  available  to  support  the  project.  This  process  is  repeated  with  all  other 
project  requirements  and  in  every  project  to  the  point  where  clear  patterns  are  followed 
without  the  knowledge  of  the  perfonners.  During  the  operations  more  patterns  are 
observed.  The  operations  such  as  surveying,  equipment  setup,  test  execution,  and  cleanup 
are  consistently  repeated  for  each  project.  Each  of  the  tasks  is  completed  per  procedures 
established  by  the  organization  or  as  per  requirements  of  the  project.  The  repeatability  of 
the  procedures  and  the  frequency  of  each  task  during  multiple  projects  results  to  observed 
patterns.  The  patterns  develop  as  these  named  tasks  are  perfonned  to  provide  solution  to 
problems  anticipated  during  test  operations.  For  instance,  surveying  is  conducted  to 
obtain  equipment  setup  locations  to  ensure  valid  data  collection.  Appropriate  equipment 
set  up  ensures  smooth  operations  with  fewer  risks.  Cleaning  up  includes  clearing  the 
range  of  all  test  equipment  and  any  hazardous  or  non-hazardous  waste  generated  during 
the  operations,  returning  equipment  for  repair  and  storage,  and  personnel  retrograde  to 
home  commands. 

Reporting 

Upon  completion  of  test  execution  and  data  validation,  reduction,  and  analysis 
reporting  phase  begins.  Reporting  includes  incorporating  the  data  analysis  results  to 
established  report  templates.  The  templates  used  are  repeatedly  used  for  different  projects 
with  slight  tailoring  to  fit  each  project’s  requirements  and  specifications.  During  this 
reporting  phase  the  following  are  accomplished:  validation  of  raw  data,  reduction  of  raw 
data,  data  analysis,  preparation  of  draft  test  report,  peer  review  of  draft  report,  completion 
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of  test  report,  publishing  of  final  report,  delivery  of  test  report  to  sponsor,  archiving  of 
test  report  for  future  reference.  Data  validation  ensures  that  the  correct  and  complete  data 
is  available.  The  data  reduction  is  perfonned  prior  to  analysis  and  includes  cleaning  up 
the  data  to  enable  easier  analysis.  Data  analysis  results  showing  are  incorporated  in  the 
test  report  and  are  used  by  the  decision  makers  to  determine  if  the  project  objectives  have 
been  met.  The  final  report  submitted  to  the  sponsor  consists  of  standard  sections  for  each 
project.  It  includes  system  description,  test  results,  conclusions,  and  recommendations. 
The  test  report  must  communicate  the  results  of  completed  tests  to  decision  authorities  in 
a  timely,  factual,  concise,  comprehensive,  and  accurate  manner  (Claxton,  2005). 

Description 

Description  is  the  second  and  last  part  of  the  pattern  development  methodology. 
Each  pattern  identified  satisfies  the  three  main  pattern  parts:  pattern  name,  problem,  and 
solution.  A  pattern  name  is  a  descriptive  and  unique  name  that  helps  in  identifying  and 
referring  to  the  pattern.  As  observed  in  Introduction  Chapter,  a  pattern  is  described  as  a 
named  and  well-known  problem/solution  pair.  In  this  thesis,  each  identified  pattern  was 
named  and  their  problem/solution  pair  identified.  The  names  were  selected  based  on  the 
kind  of  pattern  identified.  For  instance,  a  pattern  presenting  Cost  Estimating  process  was 
simply  named  Cost  Estimate  and  a  pattern  presenting  the  three  primary  phases  in  T&E 
process  was  simply  named  Primary. 

The  test  subjects  in  this  thesis  are  the  various  patterns  developed.  In  results 
section  the  patterns  are  demonstrated  in  form  of  their  structure  and  purpose.  The  structure 
shows  each  part  of  the  pattern  and  its  relationship  with  the  other  parts. 
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IV.  RESULTS 


Chapter  Overview 

This  chapter  presents  the  results  obtained  from  this  investigation.  The  results  include  the 
presentation  of  various  patterns  identified  and  associating  each  of  the  patterns  with  the 
corresponding  T&E  challenge  to  be  resolved  using  the  pattern. 

T&E  Patterns 

Based  on  T&E  operations  and  challenges  encountered  during  the  operations,  eight 
T&E  patterns  have  been  identified  and  described  in  this  thesis.  The  patterns  are  in  three 
categories  as  used  for  the  23  patterns  of  the  GoF.  These  categories  are  creational, 
behavioral,  and  structural.  In  this  thesis,  five  behavioral  patterns,  one  creational  pattern, 
and  one  structural  pattern  were  identified.  Table  4  shows  a  list  of  the  patterns  identified. 


Table  4.  T&E  Patterns 


Pattern 

Category 

Pattern 

Name 

Problem 

Solution 

Behavioral 

Primary 

What  are  major  T&E 
processes? 

Complete  the  planning,  execution,  and 
reporting  phases  in  every  T&E  project 

Objective 

What  are  project  objectives  and 
how  are  will  they  be  achieved? 

Identify  sponsor  requirements  and  conduct 
T&E  operation  to  achieve  project  objective 

Cost 

Estimating 

Are  the  project  costs  correctly 
identified  and  funds  assigned? 

Conduct  cost  estimate  based  on  sponsor 
requirements  to  determine  the  cost  for 
travel,  material,  and  labor 

Schedule 

Are  project  events  and 
documentation  completed  in  a 
timely  manner? 

Based  on  sponsor  requirements  and 
resource  availability,  determine  the 
appropriate  schedule  for  all  project  events 

Work 

Acceptance 

Process 

What  work/T&E  projects 
should  an  organization  accept? 

Review  the  Statement  of  Work  to 
determine  appropriate  capabilities  to 
accomplish  the  project  objectives 

Risk 

Management 

Are  the  projects  correctly 
managed? 

Develop  and  implement  a  risk  management 
plan 

Creational 

Hazard 

Control 

How  procedures  can  help  to 
avoid  hazards  during  operations 

Create  and  follow  a  standard  procedure  to 
control  hazards  during  operations 

Structural 

Test  Team 

Who  is  supporting  the  test? 

A  test  team  is  developed  with  appropriate 
skills  to  accomplish  test  objectives. 
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Behavioral  Patterns 


Name:  Primary  Pattern 

The  primary  T&E  operation  pattern  consists  of  the  three  main  phases  of  the  T&E 
operation.  These  phases  completed  sequentially  in  each  operation  include  planning, 
execution,  and  reporting.  Planning  encompasses  all  the  necessary  preparation  prior  to 
subsequent  processes.  As  discussed  previously,  preparation  includes  selection  of  test 
item(s),  establishment  of  test  procedures,  selection  of  test  site,  selection  and  training  of 
test  personnel,  preparation  of  test  facilities  and  resources,  the  acquisition  of  support 
equipment,  and  test  supply  support.  Execution  involves  conducting  the  actual  test  to 
detennine  system  performance.  The  main  events  during  execution  phase  include  traveling 
to  the  test  site,  setting  up  the  appropriate  data  collection  equipment,  conducting  the  test 
execution  per  the  test  plan,  and  collect  the  type  and  amount  of  required  data.  The 
reporting  phase  consists  of  data  validation,  data  reduction,  data  analysis,  and  report 
writing.  The  data  validation  is  performed  to  ensure  that  the  valid  data  was  collected. 
Reduction  is  cleaning  up  of  data  prior  to  analysis.  Once  the  data  is  analyzed  using 
statistical  methodology  and  other  appropriate  techniques,  the  results  are  incorporated  in 
the  fonnal  report  and  delivered  to  the  sponsor. 

Intent:  The  three  major  processes  in  the  T&E  effort  are  planning,  execution,  and 
reporting.  In  every  T&E  event,  these  processes  are  repeated  in  the  following  order: 
planning  — >  execution  — >  reporting. 

Also  Known  As:  Not  applicable  (n/a). 

Motivation  (forces):  As  simple  as  in  everyday  life,  human  beings  are  constantly  thinking 
first  (planning)  before  reacting  (execution)  and  getting  results  of  their  actions  (reporting). 


31 


In  the  T&E  this  pattern  is  always  performed  regardless  of  the  level  for  each  project.  The 
completion  of  the  pattern  is  an  assurance  that  the  Operation  has  been  conducted 
appropriately. 

Applicability:  This  pattern  is  applicable  in  everyday  life  in  all  activities  where  a 
reportable  successful  operation  is  expected.  All  the  T&E  events  in  DOD  requires 
planning,  execution,  and  reporting;  planning  to  ensure  successful  execution,  execution  to 
test  system  capability,  and  reporting  to  test  the  capability  of  the  system. 

Structure:  The  graphical  presentation  of  the  Primary  T&E  Operation  Pattern  is  shown  in 
Figure  10. 


Figure  10.  Primary  Pattern 

Participants:  The  participants  in  this  pattern  include  planning,  execution,  and  reporting. 

•  Planning  -  planning  is  the  first  step  in  T&E  Operation.  It  includes  all  the 
necessary  preparation  prior  to  execution. 

•  Execution  -  Test  execution  includes  executing  a  test,  collecting  data, 
conducting  data  reduction,  validation,  and  analysis,  and  conducting  a  data 
review  to  ensure  data  is  adequate  to  support  evaluation. 

•  Reporting  -  reporting  is  the  final  step  in  this  pattern.  It  is  the  step 
immediately  before  project  close-out. 
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Collaboration:  The  three  phases  collaborate  with  each  by  order  of  precedence  to 
accomplish  the  pattern’s  objective.  The  phases  are  completed  sequentially  starting  with 
planning. 

Consequences:  Using  the  pattern  leads  to  streamlined  Operation  to  efficiently  and 
effectively  conduct  the  T&E  events.  However,  as  mentioned  in  previous  section,  there  are 
tradeoffs  for  using  the  pattern. 

Implementation:  The  implementation  of  this  pattern  is  observed  in  all  projects 
undergoing  T&E  phase  of  acquisition  frame  work. 

Known  Uses:  This  is  the  primary  pattern  used  while  supporting  a  T&E  effort. 

Related  Patterns:  Related  patterns  are  the  patterns  developed  from  individual  Operations 
of  this  pattern.  This  primary  pattern  is  included  in  all  other  patterns  as  it  will  be  shown 
here. 

Name:  Objective  Pattern 

The  objective  pattern  consists  of  all  the  necessary  steps  perfonned  to  achieve  the 
objective  of  the  T&E  operation.  For  instance,  a  Service  Life  and  Accelerated  Age  test  for 
Time  Blasting  Fuse  Igniter  (M760)  was  conducted  to  update  the  Operational  Reliability 
and  Service  Life  Projection.  The  objective  for  this  test  was  to  update  the  R0  and  SLP  of 
the  item  and  was  achieved  by  using  the  objective.  The  pattern  presents  the  steps 
necessary  to  meet  the  T&E  objective  efficiently  and  effectively.  These  steps  include 
ensuring  that  complete  documentation  required  for  the  test  is  available  and  that  all  the  test 
processes  have  been  considered  and  correctly  planned  for.  For  instance,  SOW  is  first 
received  by  the  testing  agency  from  the  sponsor  to  determine  customer  requirements 


33 


(T&E  objective).  The  inputs  to  the  SOW  include  customer  requirements,  test 
requirements,  and  kick-off  meeting  guidance.  The  testing  organization  then  develops  a 
WUA  stating  the  tasks  to  be  accomplished  to  meet  the  test  objective.  The  WUA  inputs 
include  cost  estimate  and  SOW.  Additionally,  the  WUA  provides  the  estimated  cost, 
deliverables,  and  the  due  dates  for  each  deliverable.  Once  the  sponsor  signs  the  WUA, 
the  testing  agency  develops  a  test  plan  along  with  a  standard  operating  procedure  (SOP) 
to  be  used  during  the  execution  of  the  test. 

Intent :  this  pattern  is  used  to  ensure  that  the  test  objectives  are  met.  It  is  a  pattern 
composed  of  steps  necessary  to  accomplish  the  test  objective.  The  test  objective  is  to  test 
the  system  in  accordance  with  customer’s  specifications  to  detennine  to  if  the  system 
requirements  are  met. 

Also  Known  As:  n/a. 

Motivation  (Forces)’.  All  the  T&E  events  have  one  objective  in  common:  testing  system 
capability.  The  desire  to  test  the  system  per  customer’s  specifications  and  requirements 
drives  the  need  and  usage  for  this  pattern. 

Applicability :  This  pattern  is  applicable  in  all  T&E  events  conducted  for  DOD  systems. 
Structure:  The  pattern  in  Figure  1 1  shows  the  flow  process  of  achieving  the  T&E 
objective.  It  includes  the  main  processes  and  documentation. 
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Figure  11.  Objective  Pattern 

Participants:  As  described  in  the  solution  paragraph  and  as  shown  in  the  structure,  the 
participants  (as  shown  in  the  structure  above)  include  test  plan,  Op  Plan,  SOP,  Timeline, 


Test  report. 

The  test  plan  provides  detailed  information  about  the  test  objective,  system 
description,  schedule,  test  personnel  infonnation,  test  execution  procedure,  test  samples, 
and  data  forms  and  data  collection  requirements. 

The  OpPlan  is  a  planning  tool  prepared  in  SharePoint  for  tracking  the  status  of 
project  planning.  It  is  reviewed  and  approved  by  the  management  team  prior  to  test 
execution.  It  includes  project  name,  estimated  test  dates,  equipment  requirements, 
personnel  requirements,  test  samples  requirements,  final  project  documentation,  and  Op 
Plan  Checklist.  It  also  includes  a  comments  section  to  gather  comments  and  remarks 
provided  by  stakeholders  including  the  Op  Plan  reviewers  and  approvers.  The  review 
and  approval  process  is  initiated  by  the  PL  once  he  has  confirmed  that  every  action  item 
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in  the  checklist  has  been  completed.  Through  signature  flow  process,  the  Op  Plan  is 
subsequently  sent  to  all  the  reviewers  and  approvers. 

Standard  Operating  Procedure  (SOP)  is  a  safety  documentation  that  contains 
procedures  to  be  followed  during  T&E  event  for  safe  operation  to  avoid  injury  to 
personnel  or  damage  to  equipment.  It  includes  a  hazard  analysis  table  consisting  of  all  the 
identified  potential  hazards.  Prior  to  starting  of  operation,  a  safety  brief  is  conducted  to 
ensure  that  test  participants  are  aware  of  the  potential  hazards. 

In  addition  to  OpPlan,  the  PL  is  expected  to  develop  a  timeline  showing  the 
amount  of  time  each  step  of  the  T&E  Operation  takes.  For  instance,  it  will  show  the 
number  of  days  required  to  complete  preparing  a  test  plan.  It  also  shows  the  cumulated 
amount  of  time  at  every  step  of  the  process  from  planning  to  reporting. 

Collaboration:  The  objects  in  this  pattern  work  together  to  achieve  the  test  objective. 
Sometimes  the  objects  (documents)  are  references  to  each  other. 

Consequences:  Using  this  pattern  leads  to  a  standard  and  more  effective  and  efficient 
method  of  achieving  test  objectives  of  current  and  future  projects,  it  helps  plan,  execute, 
and  report  on  T&E  operations  with  less  resources  since  every  time  the  process  is 
conducted  is  reusing  already  established  method  that  has  been  used  many  times.  The  test 
team  will  spend  less  time  and  funds  during  planning,  executing,  and  reporting  for  a  new 
project  if  the  pattern  is  effectively  implemented.  Furthennore,  use  of  this  pattern  results 
to  fewer  mistakes  as  the  steps  in  the  pattern  have  been  conducted  before  and  have  been 
mastered.  Contrary,  using  the  pattern  discourages  innovation.  For  instance,  the 
organization  will  be  reluctant  to  invest  in  investigating  to  new  methods  as  long  as  the 
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current  methods  are  working  sufficiently.  This  results  to  an  organization  and  workforce 
with  no  opportunity  or  motivation  of  improving  their  process. 

Implementation:  This  pattern  is  used  to  detennine  test  objective.  It  ensures  that  effective 
method  is  employed  to  assure  successful  execution  of  T&E  operation  as  expected  by  the 
sponsor.  Proper  implementation  of  this  pattern  during  all  phases  of  T&E  operation  could 
result  to  a  more  successful  and  efficient  operation. 

Known  Uses:  This  pattern  is  used  during  the  planning,  execution,  and  reporting  of  most 
DOD  T&E  Operations. 

Related  Patterns:  The  related  are  all  the  T&E  patterns  included  in  this  methodology 
section. 

Name:  Cost  Estimate  Pattern 

Cost  Estimate  (CE)  is  conducted  upon  acceptance  of  the  rough  order  of  magnitude 
(ROM)  by  the  customer  to  detennine  the  amount  of  funds  required  to  complete  the  T&E 
operation.  The  command  approve  template  consisting  of  fonnulas  to  calculate  the  labor 
amount  using  the  correct  current  Fiscal  Year  labor  rates  is  used  to  develop  the  CE.  The 
template  consists  of  the  three  phases  of  the  T&E  operation:  Planning,  Execution, 
Reporting.  Labor,  travel,  and  material  costs  are  estimated  for  each  phase.  The  labor 
portion  is  broken  out  into  the  number  and  specialty  of  the  personnel  supporting  the 
project,  the  number  of  days  and  the  number  of  hours  per  day  each  personnel  category  will 
be  working  until  the  completion  of  the  assignment.  Additionally,  the  two  labor  types, 
government  and  contractor  support,  are  calculated  separately  to  ensure  the  appropriate 
type  of  funds  are  detennined.  The  travel  portion  consists  of  the  per  diem,  lodging,  and 
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mode  of  transportation  expenses  based  on  the  travel  requirements  and  mainly  controlled 
by  the  amount  of  travel  and  location  of  the  T&E  events.  Lastly,  the  material  and 
equipment  costs  are  included  if  any  required  for  T&E  support.  The  summary  of  funds 
showing  the  break-out  of  funds  into  labor,  travel,  and  material  for  both  government  and 
contractor  support  is  incorporated  into  the  Work  Unit  Assignment. 

Intent:  This  pattern  ensures  that  the  cost  estimate  for  the  project  is  correctly  done  to 
ensure  availability  of  funds  to  support  the  T&E  project. 

Also  Known  As:  N/A 

Motivation  (Forces):  There  always  limited  funds  to  execute  DOD  events.  Organizations 
are  always  encouraged  to  do  more  with  less.  Additionally,  when  bidding  for  T&E  project, 
usually  there  is  more  than  one  qualified  testing  agency  and  usually  the  least  expensive 
once  gets  the  work.  A  good  CE  is  the  starting  point  to  detennine  how  much  money  is 
enough,  and  not  a  penny  more,  to  accomplish  test  objective. 

Applicability:  This  pattern  is  applicable  in  all  projects  that  will  require  funds  to  conduct; 
usually  all  the  DOD  projects. 

Structure:  The  cost  estimate  template  is  used  to  determine  the  cost  for  completing  the 
project  tasks.  Figure  12  presents  the  cost  estimate  structure  showing  documentation  and 
process  requirements  to  obtain  correct  amounts  for  each  cost  element;  travel,  labor,  and 
material. 
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Figure  12.  Cost  Estimate  Pattern 

Participants:  The  participants  in  this  pattern  are  all  the  factors  considered  in  developing  a 
good  CE.  These  include  labor  hours,  travel  expenses,  and  material  expenses,  test  phases 
(planning,  execution,  reporting). 

Collaboration:  Objects  collaborate  to  achieve  the  best  results.  For  instance,  they  obey  the 
precedence  rule;  planning  estimate  is  completed  prior  to  execution  estimate  while  the 
reporting  estimate  is  completed  after  the  execution  estimate.  Additionally,  the  objects 
can  give  or  take  depending  on  the  PL’s  requirements  to  ensure  a  good  estimate.  For 
instance,  if  the  travel  requires  more  than  calculated,  the  labor  object  can  be  reduced  to 
meet  the  travel  object’s  requirement. 

Consequences:  As  previously  mentioned,  using  any  pattern  has  consequences;  good  or 
bad.  Using  CE  pattern  prevents  mistakes  that  could  lead  to  underestimating  or 
overestimating  of  a  project.  Also,  it  saves  time  during  development  of  a  new  CE  usually 
what  needs  to  be  done  is  to  change  a  few  things  and  adjust  numbers  based  on  the 
requirements  of  a  new  project. 
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Implementation:  CE  pattern  is  used  in  all  projects.  CE  ensures  that  all  the  cost 
requirements  for  the  project  have  been  accounted.  A  good  CE  alleviates  the  cost 
challenges  previously  discussed.  The  risks  of  running  out  of  funds  before  the  project  is 
completed  are  significantly  reduced  when  a  good  CE  pattern  is  used  in  all  projects  and 
especially  with  new  and  unfamiliar  projects. 

Known  Uses :  This  pattern  is  used  during  the  planning  of  a  new  project.  It  is  used  to 
detennine  the  amount  of  funding  is  required  to  complete  all  the  project  tasks. 

Related  Patterns:  No  other  patterns  are  related  to  this. 

Name:  Schedule  Pattern 

The  schedule  pattern  solves  the  problem:  what  is  the  required  test  schedule  to 
complete  the  test  objectives  in  a  timely  manner?  The  solution  will  include  determining 
schedule  based  on  test  requirements,  collaborating  with  all  the  stakeholders,  developing 
an  effective  timeline,  developing  an  operations  plan  on  Microsoft  SharePoint  while 
considering  all  the  risks  to  include  scope  creep,  weather,  lack  of  funds,  insufficient 
personnel,  equipment  support,  etc.  Test  schedule  starts  when  the  project  is  initiated  and  is 
usually  adjusted  as  required  and  possibly  during  the  execution  of  T&E  operations. 
Schedule  pattern  includes  the  accounting  of  all  the  actions  and  their  assigned  execution 
durations.  The  Operations  Plan  and  timeline  are  good  resources  during  development  and 
tracking  of  schedule.  The  OpPlan  shows  all  the  action  items  to  be  completed  during 
planning  of  the  T&E  prior  to  test  execution  while  the  timeline  shows  the  start  and 
completion  times  of  all  the  steps  in  the  T&E  operation. 
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Intent :  Test  schedule  pattern  allows  easier  scheduling  of  test  functions  by  following  an 
established  and  repeatable  process.  Scheduling  pattern  ensures  that  the  organization  uses 
a  known  process  of  scheduling  that  guarantees  completion  of  projects  tasks  in  a  timely 
manner. 

Also  Known  As:  Test  schedule  is  also  known  as  a  Period  of  Performance  because  it 
shows  the  overall  duration  the  funding  will  be  available  to  complete  the  project. 
Motivation  (Forces):  Test  scheduling  pattern  is  driven  customer  requirements,  project 
requirements,  and  resource  availability.  The  customer  usually  indicates  in  the  SOW  the 
period  they  expect  to  receive  test  deliverables  (test  plan,  test  report,  etc.).  The  test 
schedules  will  be  developed  so  as  to  meet  the  dates  specified  by  the  customer. 
Additionally,  test  scheduling  will  be  developed  based  on  the  project  conducted.  For 
instance,  scheduling  for  a  small  project  requires  less  work  than  scheduling  for  a  big  and 
complicated  project.  Finally,  the  availability  of  resources  is  a  major  factor  affecting  the 
scheduling  of  projects.  For  example,  two  tests  using  similar  resources  such  as  test 
equipment,  ranges,  personnel,  etc.,  will  be  scheduled  at  different  times  to  avoid  resource 
constraint. 

Applicability:  Scheduling  pattern  is  applicable  in  all  T&E  operations.  It  is  applied  during 
planning  phase  to  ensure  that  all  the  necessary  preparations  have  been  completed  prior  to 
execution  of  test.  During  execution  phase,  scheduling  of  events  is  at  most  importance 
because  it  ensures  a  collection  of  valid  data  for  analysis  to  detennine  system’s 
capabilities. 

Structure:  Scheduling  pattern  in  Figure  13  encompasses  the  steps  and  milestones  of  the 
project. 


41 


Figure  13.  Scheduling  Pattern 

Participants:  Participants  in  scheduling  pattern  include  resources  scheduled,  milestones, 
tasks  planned,  tasks  completed,  upcoming  tasks,  dates,  and  durations  for  task  completion. 
Collaboration:  Availability  of  resources  ensures  completion  of  planned  tasks  hence 
achievement  of  milestones  as  required  by  the  testing  agency.  The  dates  to  start  and 
complete  the  planned  tasks  will  be  shown  on  the  schedule.  Milestones  are  achieved  when 
tasks  are  completed  as  reflected  in  the  scheduling  pattern.  The  duration  for  each  task  is 
detennined  by  the  start  and  end  dates  assigned. 

Consequences:  Using  the  scheduling  pattern  ensures  that  the  major  project  actions  are 
scheduled  appropriately  to  avoid  any  delay  of  deliverables  to  the  sponsor.  The  project 
leader  saves  a  lot  of  time  that  is  used  in  completing  other  project  actions  by  using  the 
scheduling  pattern.  Additionally,  the  scheduling  pattern  provides  the  project  leader  with  a 
peace  of  mind  knowing  that  there  is  no  risk  of  schedule  slip  since  the  pattern  has  been 
successfully  used  with  past  projects. 
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Implementation:  This  pattern  is  implemented  during  all  the  phases  of  the  T&E 
operations.  The  timely  completion  of  scheduled  events  relies  on  an  effective  scheduling 
pattern.  It  is  advisable  to  use  it  all  the  time  for  consistency  and  assurance  for  success. 
Known  Uses:  Used  during  the  scheduling  of  T&E  events.  This  pattern  is  useful  when 
planning,  executing,  reporting,  and  closing  out  of  all  DOD  T&E  operations  to  avoid 
schedule  delays,  cost  overruns  due  to  schedule  slip,  and  loss  of  key  T&E  personnel  due  to 
reassignments  at  the  end  of  initially  set  period  of  perfonnance. 

Related  Patterns:  Test  schedule  pattern  works  with  the  collaboration  of  other  T&E 
patterns.  The  schedule  pattern  is  developed  upon  completion  of  Test  Objective  pattern. 

Name:  Work  Acceptance  Process  Pattern 

DOD  is  a  complex  organization  with  innumerable  systems  requiring  to  undergo 
T&E  prior  to  transitioning  to  the  warfighter.  Each  of  these  systems  has  specific  T&E 
requirements  that  require  skills  and  capabilities  to  meet.  The  Work  Acceptance  Process 
detennines  if  the  resources  and  capabilities  are  available  to  meet  the  test  requirements. 
This  is  done  by  conducting  a  work  acceptance  review  with  attendance  of  the  project 
leader,  program  manager,  and  select  organizational  management  team  members. 

Intent:  The  pattern  ensures  that  a  consistent  process  is  used  all  the  time  during  the 
acceptance  of  T&E  project. 

Also  Known  As:  n/a. 

Motivation  (Forces):  DOD  is  a  complex  system  consisting  of  many  agencies,  systems, 
and  a  large  workforce.  Among  the  agencies  are  T&E  agencies  responsible  for  testing  and 
evaluating  systems  prior  to  transitioning  to  the  warfighter.  A  successful  T&E 
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organization  is  equipped  with  capable  workforce  and  equipment  for  conducting  the  T&E 
operations.  This  pattern  is  developed  to  ensure  that  the  organization  is  well  equipped  to 
test  and  evaluate  the  systems  provided  by  the  developing  sponsor. 

Applicability:  This  pattern  is  applicable  in  all  projects  and  as  soon  the  sponsor  provides  a 
statement  of  work  to  the  testing  organization. 

Structure :  The  work  acceptance  process  is  provided  in  Figure  14. 


Figure  14.  Work  Acceptance  Process  Pattern 

Participants:  Participants  in  this  pattern  include  work  type,  customer  requirements, 
documentation  requirements,  and  review  process. 

Collaboration:  During  the  work  acceptance  process,  the  work  type  is  evaluated  by  the 
management  to  determine  if  is  supportable  by  the  organization.  The  customer 
requirements  are  related  to  the  type  of  work  received.  For  instance,  Surveillance  Testing 
will  have  customer  requirements  such  as  determining  or  updating  the  Service  Fife 
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Projection  and  Operational  Reliability.  The  review  process  is  conducted  by  referring  to 
the  documentation  requirements  and  customer  requirements. 

Consequences:  There  are  a  lot  of  benefits  for  using  this  pattern.  First,  the  right  and 
supportable  work  is  accepted  and  the  required  documentation  for  funding  is  prepared  in  a 
timely  manner  before  the  period  of  perfonnance  starts. 

Implementation:  This  pattern  is  implemented  in  the  acceptance  of  all  new  work  in  the 
organization. 

Known  Uses:  Organization  use  this  pattern  to  ensure  that  the  work  performed  is 
supportable  based  on  the  capabilities  of  the  organization. 

Related  Patterns:  n/a. 

Name:  Risk  Management  Pattern 

Risk  management  pattern  is  a  behavioral  pattern  developed  to  effectively  manage 
project  risks.  Risk  management  is  application  of  skills,  knowledge,  and  risk  management 
tools  and  techniques  to  your  projects  to  reduce  threats  to  an  acceptable  level  while 
maximizing  opportunities  (Heldman,  2012).  Hilson  and  Simon  in  (Simon,  2012) 
describes  risks  as  arising  from  uncertainty  and  they  are  about  the  impact  that  uncertainty 
events  or  circumstances  could  have  on  the  achievement  of  goals.  In  Risk  Management 
Guide  for  DOD  Acquisition  (Defense,  2006),  risk  is  defined  as  a  measure  of  future 
uncertainties  in  achieving  program  perfonnance  goals  and  objectives  within  defined  cost, 
schedule  and  performance  constraints.  This  risk  management  pattern  encompasses  all  the 
steps  conducted  to  ensure  a  successful  operation  without  impact  of  risks.  The  potential 
risks  considered  in  risk  management  during  a  T&E  operation  include  mismanagement  of 
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funds,  cost  overrun,  schedule  slip,  poor  service  or  product  quality,  dissatisfied  customers, 
damage  to  equipment,  and  injury  to  personnel. 

Intent:  This  pattern  provides  an  easier  way  to  ensure  that  the  operation  will  be  conducted 
successfully  with  reduced  risks.  Use  of  the  pattern  provides  the  ability  to  reuse  what  has 
been  used  before  and  is  believed  to  work.  DOD  testing  organizations  face  potential 
shutdown  if  they  fail  to  manage  critical  risks.  For  instance,  risks  resulting  in  death  can 
result  in  a  shutdown. 

Also  Known  As:  n/a. 

Motivation  (Forces):  The  forces  that  drive  the  development  of  risk  management  include 
fear  of  failure,  limited  funds,  limited  time,  sponsor  oversight,  critical  project,  customer 
expectations,  desire  for  success  without  failure,  and  management  support.  Due  to  the 
mentioned  forces  that  could  lead  to  loss  of  business  or  failure  to  provide  needed  support 
to  the  warfighter,  T&E  organizations  will  be  driven  to  follow  a  risk  management  plan 
that  has  worked  before  and  guarantees  success  for  current  and  future  projects. 
Applicability:  Risk  management  pattern  is  applicable  in  the  planning,  execution,  and 
reporting  of  all  projects.  It  is  also  applicable  to  both  small  and  big  projects.  Finally  it  is 
applicable  all  the  time  during  the  project  lifetime. 

Structure:  Since  identified  risks  are  always  reoccurring  during  operation  and  new  risks 
are  developed  and  identified  during  operation,  risk  management  pattern  is  best  presented 
as  an  iterative  process.  As  described  in  (Denis,  2010),  the  structure  in  Figure  15  includes 
the  following  steps: 

(1)  Risk  Strategy:  Decision  on  how  to  manage  risk  throughout  the  duration  of  the 
operation 
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(2)  Identify  Risk:  Identify  what  is  the  risk,  and  what  is  at  risk. 

(3)  Allocate  Ownership:  Owner  is  best  person  positioned  to  perform  mitigating 
actions  on  the  risk  and  monitor  the  risk. 

(4)  Evaluate  Risk:  Evaluating  and  assigning  the  risk  with  an  impact  and  probability 
score. 

(5)  Plan  Mitigations:  In  a  general  sense,  every  risk  has  a  standard  set  of  mitigations 
which  can  be  applied  to  it.  These  are  commonly  referred  to  as  ‘The  4  Ts’: 

a.  Transfer  -  transfer  the  risk  to  another  party 

b.  Tolerate  -  Do  nothing  option  for  risks  with  very  low  impact 

c.  Terminate  -  Adjusting  the  program  so  the  risk  is  no  longer  applicable  to 
the  program. 

d.  Treat  -  Take  actions  to  reduce  the  probability  and  impact  of  risk. 

(6)  Implement  Actions:  Concrete  actions  to  the  risk  owner  for  risk  management 

(7)  Monitor  and  Control:  Review  of  risks  and  actions  so  the  risk  impact  and 
probability  can  be  updated  following  any  actions  which  have  been  perfonned  to 
treat  them. 


Figure  15.  Risk  Management  Pattern 

Participants:  Participants  in  this  pattern  are  the  four  main  steps  for  risk  management  and 
the  actions  perfonned  during  each  main  step. 
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Collaboration:  As  the  pattern  structure  shows,  each  main  step  is  dependent  of  the 
completion  of  the  preceding  or  succeeding  one. 

Consequences :  Using  risk  management  pattern  could  result  in  many  benefits  that  include 
less  or  no  injuries,  cost  overruns,  schedule  slips,  poor  work  quality,  loss  of  equipment, 
and  loss  of  skilled  personnel.  Contrary,  using  the  pattern  may  result  to  increase  in 
program  costs  because  funds  will  be  required  to  ensure  consistency  use  of  the  pattern  in 
all  projects  even  in  some  would  not  require  it. 

Implementation:  This  pattern  is  implemented  in  the  planning,  execution,  reporting,  and 
closing  out  phases  of  all  projects.  The  pattern  is  repeated  in  each  phase  to  identify, 
analyze,  and  control  all  risks. 

Known  Uses:  Risk  management  pattern  is  used  all  the  time  during  risk  management 
meaning  all  the  time  throughout  the  lifetime  of  the  project. 

Related  Patterns:  Hazard  control  is  a  related  pattern  to  risk  management  pattern.  In  risk 
management,  safety  hazards  are  identified  as  some  of  the  risks  in  a  project  that  could 
result  in  increased  cost  and  schedule  slip  if  they  are  not  effectively  managed  and 
controlled. 

Creational  Pattern 
Name:  Hazard  Control 

Hazards  are  controlled  to  ensure  safe  T&E  operations.  As  it  is  impossible  to 
identify  all  the  risks  associated  with  any  project,  it  is  also  not  possible  to  identify  all  the 
hazards  in  an  operation.  Organizations  have  established  hazard  analysis  procedures  to 
identify  and  mitigate  operation  hazards.  For  instance,  in  (CRANE,  2009),  hazard 
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analysis  is  defined  as  the  process  of  identifying  and  evaluating  the  hazards  of  a  system, 
operation,  facility,  or  item  and  then  making  change  recommendations  that  would  either 
eliminate  the  hazard  or  reduce  its  risk  to  an  “acceptable  level”.  Hazard  analysis  is  a 
formal/specific  type  of  Operational  Risk  Management  (ORM)  that  is  required  for 
hazardous  operations.  The  hazard  control  pattern  represents  the  relationships  of  the  parts 
during  the  hazard  analysis  process.  Similarly  to  creational  pattern  form  in  software 
design,  the  “factory”  in  this  pattern  are  the  sources  of  information,  resources,  or  guidance 
and  the  “products”  are  the  destinations  where  the  action  is  performed  or  the  result  of  the 
guidance  provided  from  the  factory.  For  instance,  the  test  organization  as  a  “factory” 
“creates”,  as  products,  the  safety  personnel. 

Intent:  The  goal  behind  the  safety  pattern  is  to  provide  a  quicker  solution  during  the 
development  of  safety  control  plan  for  T&E  operations.  This  pattern  ensures  that  safety 
control  is  effectively  implemented  in  T&E  operations  to  avoid  hazard  mishaps. 

Also  Known  As:  n/a 

Motivation  (Forces):  This  pattern  can  be  used  during  the  review  and  approval  of  an  SOP 
for  a  conducting  T&E  of  a  new  project.  The  participants  involved  including  the  test 
organization,  SOP,  walkthrough,  and  pilot  test  would  collaborate  to  ensure  that  the  SOP 
is  completed  and  ready  for  use  during  the  operation. 

Applicability:  This  pattern  is  applicable  during  the  planning  and  execution  phases  the 
T&E  operation.  For  instance,  during  the  planning  phase,  the  SOP  is  developed  and  during 
execution  the  safety  personnel  complete  the  hazard  brief  requirements  prior  to  execution. 
Structure:  Figure  16  depicts  the  hazard  control  structure.  The  structure  shows  the 
participants  and  their  relationships  with  each  other. 
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Figure  16.  Hazard  Control 


Participants:  All  the  participants  of  this  pattern  are  shown  on  the  structure.  The  test 
organization  is  the  main  participant. 

Collaboration:  Collaboration  is  reflected  in  the  relationship  among  the  participants.  For 
instance,  the  safety  personnel  provide  hazard  brief  forms  to  test  team  to  sign  after 
completing  the  hazards  briefing. 

Consequences:  This  pattern  provides  an  easier  and  effective  way  of  ensuring  that 
hazards  are  controlled  during  operations.  There  are  no  known  side  effects  or  tradeoffs 
known  for  using  this  pattern. 

Implementation:  Some  of  the  T&E  challenges  identified  including  safety  hazards, 
schedule,  and  government  policies  are  greatly  alleviated  by  using  this  pattern. 

Known  Uses:  Used  during  the  approval  of  new  SOPs  and  execution  of  T&E  operations. 
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Related  Patterns:  Risk  management  is  a  related  pattern  hazard  control  pattern.  In  risk 
management,  safety  hazards  are  identified  as  some  of  the  risks  in  a  project  that  could 
result  in  increased  cost  and  schedule  slip  if  they  are  not  effectively  managed  and 
controlled. 

Structural  Pattern 

Name:  Test  Team 

Test  Team  is  a  kind  of  structure  pattern  consisting  of  the  personnel  required  to 
support  a  certain  test.  The  review  of  past  projects  have  provided  the  insight  to  general 
structure  of  a  test  team  pattern  that  organizations  tend  to  use  when  assigning  personnel  to 
different  roles  of  a  project.  The  test  team  structure  represents  the  key  stakeholders  of  a 
test  organization  to  ensure  successful  completion  of  project  requirements. 

Intent:  The  purpose  of  this  pattern  is  to  ensure  that  the  appropriate  personnel  are 
available  to  support  all  the  tasks  required  to  accomplish  the  project  objectives. 

Also  Known  As:  n/a. 

Motivation  (Forces):  The  motivation  forces  include  personnel  experience  level, 
personnel  availability,  personnel  skills,  test  requirements,  and  collaboration  of  personnel 
during  the  period  of  performance. 

Applicability:  This  pattern  is  applied  all  the  time  during  the  assignment  of  new  projects 
to  personnel. 

Structure:  The  structure  in  Figure  17  represents  the  Test  Team  pattern.  It  shows  the  test 
team  hierarchy  beginning  with  the  test  division’s  manager. 
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Figure  17.  Test  Team 


Participants:  participants  of  this  pattern  are  all  the  test  team  personnel  shown  at  different 
levels  of  the  structure. 

Collaboration:  All  the  four  levels  in  this  pattern  collaborate  to  achieve  project’s 
objectives.  The  relationship  among  the  levels  forces  the  participants  in  each  level  to 
collaborate. 

Consequences:  Failure  to  use  the  pattern  could  result  to  poor  management  of  personnel 
or  personnel  skills  during  project  execution.  For  instance,  unskilled  personnel  could  be 
assigned  to  wrong  tasks  if  the  pattern  is  not  used  to  identify  the  personnel  with  the 
appropriate  skills. 

Implementation  -.  This  pattern  is  implemented  during  the  planning  of  a  new  project.  The 
test  team  is  assembled  following  the  pattern  that  will  be  used  to  completed  project  tasks. 
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Known  Uses:  Used  by  organizations  when  forming  test  teams  for  assigned  projects. 
Related  Patterns:  n/a. 


Summary 

Three  categories  of  patterns  were  identified  and  described  during  this 
investigation.  The  three  categories  are  behavioral,  creational,  and  structural.  The  four 
patterns  in  behavioral  category  are  primary,  objective,  cost  estimate,  and  schedule.  Risk 
management  is  in  creational  category,  and  the  test  team  is  in  structural  category.  The 
patterns  were  identified  using  observational  methodology  that  included  reviewing  the 
documentation  of  past  and  current  projects  familiar  to  the  author.  The  eight  patterns 
which  are  precisely  described  would  benefit  the  T&E  communities  if  implemented  in 
their  processes.  Table  5  shows  the  pattern  and  corresponding  challenges  anticipated  to  be 
resolved  if  patterns  are  appropriately  implemented. 


Table  5.  Patterns  and  Challenges  in  T&E  Operations 


Pattern  Name 

Challenges 

Primary 

Program  Management 

Objective 

Data  Collection,  Cost,  Schedule 

Cost  Estimate 

Cost,  Workforce,  Test  Equipment,  Test  Facilities 

Schedule 

Cost,  Schedule,  Test  Equipment, 

Risk  Management 

Cost,  Hazards,  Schedule,  Workforce,  Test  Equipment,  DOD 
Policies,  Weather,  Data  Collection,  Collaboration,  Test  Facilities, 
Program  Management 

Work  Acceptance 
Process 

Cost,  Schedule,  Test  Equipment,  Workforce,  Test  Facilities 

Hazard  Control 

Cost,  Hazards,  Schedule,  Workforce,  Test  Equipment,  DOD 
Policies 

Test  Team 

Workforce,  Schedule,  Program  Management 
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Investigative  Questions  Answered 

The  three  investigative  questions  answered  by  this  research  were  (1)  How  to 
identify  some  of  the  challenges  encountered  by  a  typical  T&E  organization?  (2)  How  to 
detennine  some  of  the  T&E  patterns  organizations  can  use  to  plan,  execute,  and  report 
before,  during,  and  after  T&E  event?  and  (3)  How  could  the  T&E  communities  use  the 
patterns  to  overcome  challenges  encountered  during  T&E  operations?  The  first  question 
was  answered  by  identifying  the  T&E  challenges  through  experience  gained  from 
working  in  a  T&E  organization  and  review  of  documentation  of  past  reports.  A 
methodology  was  developed  that  was  used  to  develop  the  patterns  a  T&E  organization 
would  use  to  plan,  execute,  and  report  before,  during,  and  after  T&E  event.  The  patterns 
were  clearly  identified  and  described  in  the  simplest  way  possible  by  including  only  the 
necessary  steps.  The  third  question  is  about  the  implementation  and  use  of  the  patterns  to 
resolve  T&E  challenges.  The  implementation  and  known  uses  for  each  pattern  are 
provided  in  pattern  descriptions. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


Chapter  Overview 

This  is  the  final  chapter  of  the  research  and  it  presents  the  conclusion  of  research, 
significance  of  research,  recommendations  for  actions  and  recommendations  for  future 
research. 

Conclusions  of  Research 

Eight  patterns  observed  in  the  T&E  community  were  identified  and  described  in 
this  research.  The  patterns  were  presented  and  described  using  the  fonns  frequently  used 
in  the  software  community.  Through  literature  review,  it  was  detennined  that  patterns  are 
mostly  developed  and  utilized  in  software  design  than  in  hardware  or  process  design.  The 
main  purpose  for  each  of  the  patterns  identified  in  this  research  is  to  achieve  T&E 
objectives  in  the  most  effective  and  efficient  way  possible.  Consequently,  the  patterns 
developed  here  have  a  lot  of  similarity  and  dependence  to  each  other.  Finally,  this  thesis 
provides  the  answers  to  the  three  research  questions  investigated. 

An  infonnal  survey  was  presented  to  co-workers  of  varying  skill  and  experience 
levels  for  their  feedback  on  use  of  patterns  and  response  to  challenges  they  face  in  their 
projects.  The  survey  was  designed  with  a  Yes/No  response  options.  The  personnel  were 
requested  to  review  the  pattern  structures  and  T&E  challenges  before  responding  to  the 
questionnaire.  The  five  statements  below  indicate  responses  to  the  use  of  patterns  and 
impact  of  challenges  faced  in  T&E  community.  On  average,  the  personnel  agreed  that: 

1.  The  challenges  identified  really  exist  in  the  T&E  community 
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2.  The  patterns  present  the  processes  they  use 

3.  Reducing  the  occurrence  and  impact  of  T&E  challenges  would  be  more  likely 
by  using  patterns. 

4.  They  would  use  the  patterns  to  resolve  the  identified  T&E  challenges 

5.  The  difficulty  of  their  T&E  tasks  is  greatly  alleviated  by  using  the  patterns 

Significance  of  Research 

This  research  identified  and  described  patterns  used  in  the  T&E  communities. 
These  patterns  are  used  during  the  execution  of  T&E  objectives.  If  appropriately 
implemented,  these  patterns  would  help  alleviate  some  of  the  commonly  encountered 
challenges  in  the  T&E  environment  such  as  poor  project  management,  schedule  conflicts, 
cost  overruns,  strict  DoD  policies,  lack  of  appropriate  test  equipment  and  facilities,  and 
collection  of  wrong  data.  Failure  to  control  or  eliminate  the  T&E  challenges  could  result 
in  serious  impact  to  cost,  schedule,  and  quality  of  the  deliverables. 

Recommendations  for  Action 

It  is  recommended  that  all  the  DoD  T&E  communities  develop  and  use  patterns 
for  T&E  processes.  The  DOD  T&E  communities  are  encouraged  to  tailor  the  patterns 
developed  as  necessary  to  fit  their  T&E  requirements. 

Recommendations  for  Future  Research 

Compared  to  software  engineering,  there  has  not  been  much  research  in  systems 
engineering  processes  and  hardware  patterns.  In  addition  to  patterns  identified,  there  are 
more  that  can  be  identified  and  described  in  T&E  processes.  Future  research  is 
recommended  to  identify  more  T&E  patterns. 
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Summary 

This  research  provided  an  opportunity  to  identify  and  describe  patterns  in  the 
T&E  operations.  Eight  patterns  including  primary,  objective,  cost  estimate,  schedule,  risk 
management,  work  acceptance  process,  and  hazard  control,  and  test  team  were  identified. 
Additionally,  challenges  commonly  experienced  during  the  planning,  execution,  and 
reporting  of  T&E  operations  were  also  identified.  These  challenges  include  Cost, 
Hazards,  Schedule,  Workforce,  Test  Equipment,  DOD  Policies,  Weather,  Data 
Collection,  Collaboration,  Test  Facilities,  and  Program  Management.  Finally,  each 
pattern  was  associated  with  the  challenges  that  would  be  resolved  by  implementation  of 
the  pattern. 
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Appendix  A:  Acronyms 


AFIT  Air  Force  Institute  of  Technology 

CE  Cost  Estimate 

DAU  Defense  Acquisition  University 

DOD  Department  of  Defense 

DT&E  Developmental  Testing  and  Evaluation 

JCIDS  Joint  Capabilities  Integration  Development  System 

LFT&E  Life-Fire  Testing  and  Evaluation 

MOE  Measure  of  Effectiveness 

MOP  Measure  of  Performance 

MOS  Measure  of  Suitability 

n/a  Not  Applicable 

OpPlan  Operations  Plan 

OT&E  Operational  Testing  and  Evaluation 

PoP  Period  of  Performance 

Ro  Operational  Reliability 

SECNAVINST  Secretary  of  the  Navy  Instruction 

SEP  Systems  Evaluation  Plan 

SLP  Service  Life  Projection 

SME  Subject  Matter  Expert 

SOP  Standard  Operating  Procedure 

SOW  Statement  of  W ork 

T&E  Test  and  Evaluation 

TAD  Temporary  Assignment  of  Duty 

WAP  Work  Acceptance  Process 

WAR  Work  Acceptance  Review 

WUA  Work  Unit  Assignment 
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